Systems and methods for encoding alternative streams of video for use in adaptive bitrate streaming

ABSTRACT

Systems and methods for encoding alternative streams for use in adaptive bitrate streaming based upon the delay of each stream in accordance with embodiments of the invention are described. One embodiment of the invention includes memory, and a processor configured by a source encoding application to: receive multimedia content, where the multimedia content includes source video data having a primary resolution and a primary sample aspect ratio; and encode the source video data as a set of alternative video streams, where: the alternative video streams have different maximum bitrates; and the alternative video streams are encoded to have an upper bound seek delay that is equal to or less than the upper bound seek delay of streams in the set of alternative video streams that are encoded at a higher maximum bitrate.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional. Application Ser.No. 61/430,502, filed Jan. 6, 2011, the entirety of which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to adaptive bitrate streamingand more specifically to the buffering of media by playback devices inadaptive bitrate streaming systems.

BACKGROUND

The term streaming media describes the playback of media on a playbackdevice, where the media is stored on a server and continuously sent tothe playback device over a network during playback. Typically, theplayback device stores a sufficient quantity of media in a buffer at anygiven time during playback to prevent disruption of playback due to theplayback device completing playback of all the buffered media prior toreceipt of the next portion of media. Adaptive bitrate streaming oradaptive streaming involves detecting the present streaming conditions(e.g. the user's network bandwidth) in real time and adjusting thequality of the streamed media accordingly. Typically, the source mediais encoded at multiple bit rates and the playback device or clientswitches between streaming the different encodings depending onavailable resources.

A common goal with adaptive bitrate streaming is to stream the highestbitrate stream available given the streaming conditions experienced bythe playback device without stalls in the playback of media due tounderflow. Underflow occurs when the playback device receives streamingmedia at a lower speed than the speed at which the media is played back.The video used in most adaptive bitrate streaming systems is encodedusing variable bit rate encoding, which is typically more efficient.Even though the bitrate of the stream varies in time, the stream istypically described based upon its average bit rate. When variablebitrate encoding is used, the maximum bitrate of the stream is the ratethat ensures no underflow will occur given a certain buffer size. Mostplayback devices accommodate variation in the size of the encoded framesusing a buffer. In the context of video, the buffering delay (which canalso be referred to as the seek delay) is the time a playback devicemust wait between starting filling the buffer and commencing playback toprevent underflow (i.e. a certain amount of data is buffered beforedecoding can start).

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiments of the inventionencode alternative streams for use in adaptive bitrate streaming basedupon the delay of each stream. One embodiment of the invention includesmemory, and a processor configured by a source encoding application to:receive multimedia content, where the multimedia content includes sourcevideo data having a primary resolution and a primary sample aspectratio; and encode the source video data as a set of alternative videostreams, where: the alternative video streams have different maximumbitrates; and the alternative video streams are encoded to have an upperbound seek delay that is equal to or less than the upper bound seekdelay of streams in the set of alternative video streams that areencoded at a higher maximum bitrate.

In a further embodiment, the source encoding application furtherconfigures the processor to encode the alternative video streams atpredetermined maximum bitrates and maximum buffer sizes, and the upperbound seek delay of a stream is determined by dividing the predeterminedmaximum buffer size of the stream by the predetermined maximum bitrateof the stream.

In another embodiment, the source encoding application furtherconfigures the processor to encode the alternative video streams so thateach stream in the set of alternative streams has the same aspect ratio.

In a still further embodiment, the source encoding application furtherconfigures the processor to determine a lower bound seek delay for eachstream in the set of alternative video streams.

In still another embodiment, the source encoding application furtherconfigures the processor to include within each stream its lower boundseek delay.

In a yet further embodiment, the source encoding application furtherconfigures the processor to determine the alternative stream that hasthe highest lower bound seek delay, and include the highest lower boundseek delay within each stream.

In yet another embodiment, the alternative video streams are encoded inaccordance with the H.264 standard.

In a further embodiment again, the source encoding application furtherconfigures the processor to include the highest lower bound seek delayin a initial_cpb_removal_delay field in an SEI buffering message withineach stream.

In another embodiment again, the source encoding application configuresthe processor to determine the lower bound seek delay as follows:

${Delay} = {\max\;{\forall{{chunks}\left( {\frac{\sum\limits_{i = 0}^{n}{{size}\mspace{14mu}{chunk}_{i}}}{rate} - {{\left( {n - 1} \right) \cdot {duration}}\mspace{14mu}{chunk}}} \right)}}}$

In a further additional embodiment, the source encoding applicationconfigures the processor to determine the upper bound seek delay asfollows:

${Delay} = \frac{{{level}\mspace{14mu}{buffersize}} + {{level}\mspace{14mu}{{rate} \cdot {duration}}\mspace{14mu}{chunk}}}{rate}$

where level buffersize is the maximum buffer size constraint used inencoding the video stream;

-   -   duration chunk is the time taken to playback a single chunk of        video;    -   level rate is the maximum bitrate of the video; and    -   rate is the network data rate.

Another additional embodiment also includes receiving multimediacontent, where the multimedia content includes source video data havinga primary resolution and a primary sample aspect ratio using a sourceencoder, encoding the source video data as a set of alternative videostreams using the source encoder, where: the alternative video streamshave different maximum bitrates, and the alternative video streams areencoded to have an upper bound seek delay that is equal to or less thanthe upper bound seek delay of streams in the set of alternative videostreams that are encoded at a higher maximum bitrate.

A still yet further embodiment also includes encoding the alternativevideo streams at predetermined maximum bitrates and maximum buffer sizesusing the source encoder. In addition, the upper bound seek delay of astream is determined by dividing the predetermined maximum buffer sizeof the stream by the predetermined maximum bitrate of the stream.

In still yet another embodiment, also includes encoding the alternativevideo streams so that each stream in the set of alternative streams hasthe same aspect ratio.

A still further embodiment again also includes determining a lower boundseek delay for each stream in the set of alternative video streams.

Still another embodiment again also includes including within eachstream its lower bound seek delay.

A still further additional embodiment also includes determining thealternative stream that has the highest lower bound seek delay, andincluding the highest lower bound seek delay within each stream.

In still another additional embodiment, the alternative video streamsare encoded in accordance with the H.264 standard.

A yet further embodiment again also involves including the highest lowerbound seek delay in a initial_cpb_removal_delay field in an SEIbuffering message within each stream.

Yet another embodiment again also includes determining the lower boundseek delay as follows:

${Delay} = {\max{\forall{{chunks}\left( {\frac{\sum\limits_{i = 0}^{n}{{size}\mspace{14mu}{chunk}_{i}}}{rate} - {{\left( {n - 1} \right) \cdot {duration}}\mspace{14mu}{chunk}}} \right)}}}$

A yet further additional embodiment includes determining the upper boundseek delay as follows:

${Delay} = \frac{{{level}\mspace{14mu}{buffersize}} + {{level}\mspace{14mu}{{rate} \cdot {duration}}\mspace{14mu}{chunk}}}{rate}$

where level buffersize is the maximum buffer size constraint used inencoding the video stream;

-   -   duration chunk is the time taken to playback a single chunk of        video;    -   level rate is the maximum bitrate of the video; and    -   rate is the network data rate.

A further additional embodiment again includes a machine readable mediumcontaining processor instructions, where execution of the instructionsby the processor causes the processor to perform a process includingreceiving multimedia content, where the multimedia content includessource video data having a primary resolution and a primary sampleaspect ratio, and encoding the source video data as a set of alternativevideo streams, where: the alternative video streams have differentmaximum bitrates; and the alternative video streams are encoded to havean upper bound seek delay that is equal to or less than the upper boundseek delay of streams in the set of alternative video streams that areencoded at a higher maximum bitrate.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is an architecture diagram of an adaptive bitrate streamingsystem in accordance with an embodiment of the invention.

FIG. 1B is an architecture diagram of a source encoder in accordancewith an embodiment of the invention.

FIG. 1C is an architecture diagram of a playback device in accordancewith an embodiment of the invention.

FIG. 2 conceptually illustrates a process for measuring channel datarate in accordance with an embodiment of the invention.

FIG. 3 is a flow chart illustrating a process for encoding alternativestreams of video in accordance with an embodiment of the invention.

FIG. 4 is a flow chart illustrating a process for switching betweenstreams when the amount of media buffered before commencement ofplayback is determined using the upper bound seek delay in accordancewith an embodiment of the invention.

FIG. 5 illustrates a process for performing adaptive bitrate streamingin accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF DRAWINGS

Turning now to the drawings, systems and methods for encodingalternative streams for use in adaptive bitrate streaming based upon thedelay of each stream in accordance with embodiments of the invention areillustrated. The delay of a stream is the amount of time media isbuffered prior to commencing playback so that underflow does not occurduring playback. In several embodiments, an upper bound seek delay canbe determined based upon the maximum bitrate and maximum buffer sizewith which a stream is encoded. In many embodiments, every stream isencoded to have an upper bound seek delay that is equal to or less thanthe upper bound seek delay of higher bitrate streams. In severalembodiments, the upper bound seek delay of a stream is limited basedupon a predetermined maximum buffer size that is determined based uponthe size of the available buffer on common playback devices. When thestreams are encoded according to these constraints, a playback devicebuffering an amount of media prior to playback having a duration equalto the upper bound seek delay (plus a safety margin) has sufficient timeto respond to a reduction in channel data rate so that the playbackdevice can automatically switch to the optimal stream for the newchannel conditions.

In a number of embodiments, a lower bound seek delay is determined byanalyzing a specific stream and determining a minimum amount of seekdelay to prevent underflow. Utilizing the lower bound seek delay cansignificantly reduce delays when commencing playback of streaming media.However, buffering less media can mean that the playback device has lesstime to switch to a lower bitrate stream in response to a sudden drop inchannel rate. In several embodiments, the playback device responds to asudden drop in channel rate by selecting the highest bitrate stream forwhich the lower bound seek delay can be buffered within the playbackduration of the currently buffered media at the new channel rate. Inmany embodiments, safety margins are also utilized in the calculationsto accommodate variations that occur in real world systems.

The lower bound seek delay can vary depending upon the point within thestream at which playback commences. In addition, the lower bound seekdelay can vary depending upon the rate of each stream. In a number ofembodiments, the playback device obtains the lower bound seek delays forall of the streams from the server. In several embodiments, the playbackdevice only obtains the largest of the lower bound seek delays for eachstream from the server. When video is encoded in accordance with theH.264 standard developed by the Telecommunication StandardizationSector's Video Coding Experts Group together with the InternationalOrganization for Standardization (ISO), International Electro technicalCommission (IEC) and Moving Picture Experts Group (MPEG), playbackdevices in accordance with many embodiments of the invention considerthe value of the initial_cpb_removal_delay field in an SEI bufferingmessage within the H.264 stream to be equivalent to the lower bound seekdelay. In a number of embodiments, the value in theinitial_cpb_removal_delay field in each stream is modified by theadaptive bitrate streaming system to be the maximum lower bound seekdelay and/or initial_cpb_removal_delay value for all of the alternativestreams hosted by the server.

In several embodiments, the lower bound seek delay is used by theplayback device to determine the amount of media to buffer prior tocommencing playback. However, the playback device selects a streamhaving a maximum bitrate that is significantly below the channel rate sothat the buffer quickly reaches the upper bound seek delay for thestream. When the amount of media in the buffer has a playback durationequal to the upper bound seek delay, the playback device can then freelyswitch to the optimal stream for the channel conditions from that pointonward. Adaptive bitrate streaming systems, playback devices, and thebuffering and playback of adaptive bitrate streaming media in accordancewith embodiments of the invention are discussed further below.

Adaptive Streaming System Architecture

An adaptive bitrate streaming system in accordance with an embodiment ofthe invention is illustrated in FIG. 1A. The adaptive bitrate streamingsystem 10 includes a source encoding server 12 configured to encodesource media as a number of alternative streams. As is discussed furtherbelow, the source encoding server 12 generates a top level index to aplurality of container files containing the encoded streams, at least aplurality of which are alternative streams of encoded video. These filesare uploaded to a content server 14. A variety of playback devices 18can then request portions of the encoded streams from the content server14 via a network 16 such as the Internet.

In many embodiments, the top level index file is a SMIL file and themedia is stored in Matroska container files and playback devices requestportions of the files using HTTP or another similar stateless protocol.However, any appropriate index file format and/or container file formatcan be utilized to store and index encoded streams and any suitablecommunication protocol can be utilized by playback devices 18 to obtaindata from the content server 14 in accordance with embodiments of theinvention.

Although a specific architecture is shown in FIG. 1A any of a variety ofarchitectures can be utilized to perform adaptive bitrate streaming inaccordance with embodiments of the invention. Source encoders andplayback devices in accordance with embodiments of the invention arediscussed further below.

Source Encoders

In the illustrated embodiment, the adaptive bitrate streaming systemincludes one or more source encoders capable of encoding a source streamof video content into alternative streams of encoded video havingdifferent maximum bitrates and where the upper bound seek delays of eachof the streams is equal to or less than the upper bound seek delays ofany of the other alternative streams encoded at a higher bitrate. Inmany embodiments, the source encoder can be implemented using any devicecapable of encoding streams of multimedia, where the streams are encodedat different resolutions, sampling rates, upper bound seek delays and/ormaximum bitrates. The basic architecture of an adaptive streaming systemsource encoder in accordance with an embodiment of the invention isillustrated in FIG. 1B. The adaptive bitrate streaming system 30includes a processor 32 in communication with non-volatile memory 34 andvolatile memory 36. In the illustrated embodiment, the volatile memoryincludes a source encoding application 38 and alternative streams ofvideo data 40. The source encoding application 38 takes media includinga source video stream as an input and encodes the source video as aplurality of alternative streams. In many embodiments, the alternativestreams are encoded with different resolutions, sample aspect ratios,and/or maximum bitrates. Due to the differences in the maximum bitratesat which the alternative streams are encoded, a playback device canswitch between the alternative streams to perform adaptive bitratestreaming of the encoded source video. As is discussed further below, aplayback device can seamlessly down switch between streams when each ofthe alternative streams is encoded so that its upper bound seek delay isless than the playback duration of the amount of media in the buffer.When the amount of buffered media has a playback duration equal to theupper bound seek delay of any of the other alternative streams encodedat a lower maximum bitrate, then the playback device will seamlesslydown switch. As is also discussed below, the lower bound seek delay canalso be useful to playback devices in enabling more rapid commencementof playback. In several embodiments, information concerning the upperbound seek delay of a stream and/or the lower bound seek delay of streamcan be made available to playback devices by the source encodingapplication 38 by including the information in a container filecontaining the stream or by including the information in a top levelindex file that describes the stream.

In embodiments where the alternative streams of video data are encodedin accordance with the H.264 standard, the source encoding application38 can write a value indicative of the lower bound seek delay into aninitial_cpb_removal_delay field in an SEI buffering message within eachalternative H.264 stream. In many embodiments, the source encodingapplication 38 writes the lower bound seek delay and/or theinitial_cpb_removal_delay value of the alternative stream having thehighest lower bound seek delay to the initial_cpb_removal_delay field ineach of the alternative streams. As is discussed below, the maximumlower bound seek delay is typically sufficient for performing streamswitching in accordance with many embodiments of the invention. In otherembodiments, any of a variety of mechanism for including lower boundseek delay information in video streams including streams that areencoded in formats other than in accordance with the H.264 standard canbe utilized.

A process for encoding source video as a plurality of alternativestreams in accordance with embodiments of the invention is illustratedin FIG. 3. The process 80 includes receiving (82) source video andobtaining (84) maximum bitrates and maximum buffer sizes for each of thealternative video streams that will be generated by the source encoder.The alternative video streams are then encoded (86) and are typicallywritten to container files. In writing the encoded video streams tocontainer files, the process 80 can also write (88) informationconcerning lower bound seek delays into each of the alternative streamsin a manner similar to that outlined above. In many embodiments,information indicative of the lower bound seek delay of each stream isincluded with the stream. In several embodiments, the highest lowerbound seek delay of all of the streams is included in each of the files.

Although a specific architecture is illustrated in FIG. 1B, any of avariety of architectures including architectures where the applicationis located on disk or some other form of storage and is loaded intovolatile memory at runtime can be utilized to implement source encoderscapable of encoding a stream of source video content into alternativestreams of encoded video having different maximum bitrates and where theupper bound seek delays of each of the streams is equal to or less thanthe upper bound seek delays of any of the other alternative streamsencoded at a higher bitrate in accordance with embodiments of theinvention.

Playback Devices

In the system illustrated in FIG. 1A, the playback devices are personalcomputers, and mobile phones. In other embodiments, playback devices caninclude consumer electronics devices such as DVD players, Blu-rayplayers, televisions, set top boxes, video game consoles, tablets, andother devices that are capable of connecting to a server and playingback encoded media. A basic architecture for a playback device inaccordance with an embodiment of the invention is illustrated in FIG.1C. The playback device 50 includes a processor 52 in communication withnon-volatile memory 54 and volatile memory 56. In the illustratedembodiment, the non-volatile memory includes a client application 58that configures the processor and the playback device to stream contentfrom a remote server.

During operation, the client application 58 can configure the playbackdevice to purchase content from a content store and request a top levelindex file 60 from a content server or another server. When the toplevel index file 60 is received from the content server, the clientapplication 58 configures the playback device to store the top levelindex file 60 in memory 56. The client application 58 can then configurethe playback device 50 to playback content by using the top level indexfile to request portions of container files containing content from acontent server. The specific container files selected by the clientapplication 58 can be determined based upon the information available tothe client application within the top level index file 60. The requestedportions of the container files containing content are stored within abuffer 62 in the memory 56 of the playback device. The size of thebuffer typically depends upon the video buffer verifier specified for aparticular stream in the top level index file 60. Although in manyembodiments, the buffer size is fixed or determined based on any numberof factors appropriate to a specific application and/or stream(s). Theclient application can continue to request and playback content fromdifferent container files depending upon the prevailing streamingconditions until a user instruction suspending playback is received oruntil playback completes.

When the network data rate drops, the playback device can determine anappropriate lower bitrate stream based upon the amount of data that canbe downloaded from the lower bitrate stream during the time it will takefor the client application to playback the content stored within thebuffer 62 and the available data rate. Buffer underflow will not occurfollowing the stream switch provided a sufficient amount of data isdownloaded. In several embodiments, the client application 58 respondsto a sudden drop in channel rate by selecting the highest bitrate streamfor which the lower bound seek delay can be buffered within the playbackduration of the currently buffered media at the new channel rate. Whenthe playback device has already buffered an amount of video with aplayback duration equal to the upper bound seek delay of the streambeing played back and the alternative video streams are encoded in themanner outlined above, the playback device can respond to a drop innetwork data rate by simply selecting the stream that has the highestmaximum bitrate less than the new network data rate. In manyembodiments, safety margins are also utilized in the calculations toaccommodate variations that occur in real world systems. In severalembodiments, additional margins are provided to enable the playbackdevice to grow the amount of buffered data to correspond to the upperbound seek delay of the stream. The manner in which playback devicesmake stream switching decisions in accordance with embodiments of theinvention is discussed further below.

Although a specific playback device architecture is illustrated in FIG.1C, any of a variety of architectures including architectures where theclient application is located on disk or some other form of storage andis loaded into volatile memory at runtime can be utilized to implementplayback devices for use in adaptive bitrate streaming systems inaccordance with embodiments of the invention.

Measuring Channel Data Rate

Transmission of data from a content server to a playback device can beconsidered to involve a constant rate channel that varies due toexternal conditions. Playback devices in accordance with embodiments ofthe invention measure the rate of the channel and then start streamingthe encoded stream that has a maximum bitrate closest to and less thanthe measured rate of the channel (less any safety margin). In a numberof embodiments, the playback device measures the rate of the channel byrequesting data and measuring the time taken to receive the requesteddata from the server.

Due to the fact that the buffer of playback device is sometimes full andtherefore cannot accept data or requests byte ranges or chunks of datafrom container files located on the content server, intervals or gapscan occur in which the playback device does not receive any data. Inaddition, the playback device can simultaneously stream content frommultiple streams. For example, a playback device can simultaneouslyrequest content from audio, video and subtitle streams. In manyembodiments, playback devices record the size, start time and durationof each downloaded chunk from each stream. These records can be utilizedto refine bandwidth measurements by determining overlapping chunks thatwere received during a time interval and any gaps during which no datawas received during the time interval. In this way, a playback device inaccordance with embodiments of the invention can determine the channeldata rate as the sum of the data downloaded during a predetermined timeinterval divided by the effective interval, which is determined bysubtracting the duration of any gaps from the predetermined timeinterval. In many instances, the chunks requested by a playback devicecorrespond to a closed group of pictures. In several embodiments, sourcevideo is encoded into a plurality of alternative streams and eachportion of the source video is encoded as a closed group of pictures ineach of the alternative streams. By aligning the intra-coded frames inthe alternative streams in this way, the playback device can switchbetween alternative streams at the completion of the playback of aclosed group of pictures. Irrespective of the stream selected at thecompletion of the playback of a closed group of pictures, the next frameof video in the selected stream is an intra-coded frame.

A process utilized to measure channel data rate in accordance with anembodiment of the invention is conceptually illustrated in FIG. 2. Inthe illustrated embodiment, a predetermined time interval is specified70 during which data is downloaded from two video chunks 71, 72 and twoaudio chunks 73, 74. During the predetermined time interval, a gap 75 orhole occurs during which data is not received by the playback device.The channel data rate is calculated as the sum of the bytes of datadownloaded during the predetermined time interval 70 from the videochunks 71, 72 and the audio chunks 73, 74 divided by the result ofsubtracted the duration of the gap 75 from the predetermined timeinterval. Although a specific process for calculating network data rateis described above, any of a variety of processes appropriate to aspecific application can be utilized to determine network data rate inaccordance with embodiments of the invention.

Adapting to Changes in Streaming Conditions

As the data rate of a channel changes, the playback device can adapt byselecting the stream that is optimally encoded for the new channelconditions. In several embodiments, each encoded stream is encoded usingvariable bitrate encoding techniques and the playback devices utilizebuffers to smooth the variations in the encoding bitrate. The bufferingof media to smooth variations in the encoded bitrate can influence aplayback device's decision to switch between streams. When a playbackdevice buffers a sufficiently large amount of data, the playback devicehas sufficient time to switch between streams and the playback devicecan choose the highest bitrate stream that has a maximum bitrate, whichis less than the new channel rate. When the amount of buffered video hasa playback duration that is less than the time required to download aportion of video having a playback duration equal to the upper boundseek delay from a stream having the highest maximum bitrate less thanthe network data rate, the playback device may select a stream having alower maximum bitrate to prevent buffer under-flow following the streamswitch.

Reducing the amount of data that is initially buffered prior to playbackcan increase the speed with which playback commences. Buffering lessmedia, however can result in a playback device having less time toswitch between streams in the event that the channel rate suddenlydecreases. Therefore, a playback device may be forced to select a streamhaving a bitrate that is lower than would otherwise be the case to avoidunderflow. Playback devices in accordance with embodiments of theinvention can use either approach or a combination of both approaches.The manner in which playback devices in accordance with embodiments ofthe invention determine the amount of data to buffer and how to switchstreams in response to a sudden decrease in channel rate is discussedfurther below.

Buffering Variable Bit Rate Streams

The buffering of variable bitrate encoded media prior to thecommencement of playback introduces what can be referred to as abuffering or seek delay. The delay is typically equal to the time takento download a sufficient amount of media that the buffer does notunderflow, once playback commences. Assuming channel rate does notchange, a playback device should experience uninterrupted playbackfollowing the initial seek delay.

Buffering and Playback Using Upper Bound Seek Delay

In many embodiments, each stream is encoded with a maximum bitrate and amaximum buffer size. Therefore, the seek delay for a stream can bedetermined as the maximum buffer size of the stream divided by themaximum bitrate of the stream. Calculating the seek delay based upon themaximum bitrate of the stream, represents an upper bound on the amountof buffering required to commence playback and can be referred to as theupper bound seek delay. A playback device can safely commence playbackof a stream having a maximum bitrate that is less than the channel ratewhen the playback device has downloaded a portion of media having aduration equal to the upper bound seek delay or a portion of media equalin size to the maximum buffer size attribute of the stream. Afterplayback commences, on average a portion of media corresponding to theduration of the upper bound seek delay remains in the buffer as a resultof the encoding of the stream and the rate of the channel.

When the rate of the channel decreases significantly, a playback devicein accordance with an embodiment of the invention can switch to a lowerbitrate stream (i.e. a stream having a lower maximum bitrate). If theplayback device does not buffer a portion of media having a playbackduration equal to the upper bound seek delay of the lower bitrate streamprior to completing playback of the previously buffered media, then theplayback of the lower bitrate stream may stall. Given that the playbackdevice has buffered a portion of media having a playback duration equalto or greater than the upper bound seek delay, underflow can be avoidedif the upper bound seek time of the lower bitrate stream is equal to orless than the upper bound seek delay of the higher bitrate stream.Accordingly, if every stream is encoded to have an upper bound seekdelay that is equal to or less than the upper bound seek delay of higherbitrate streams, then the playback device can smoothly transitionbetween streams that are optimal for given channel conditions withoutregard to seek delay.

Switching Between Streams

A process for switching between streams when the amount of mediabuffered before commencement of playback is determined using the upperbound seek delay is illustrated in FIG. 4. The process 100 includesselecting (102) the optimal stream for the playback conditions andbuffering a portion of media having a playback duration equal to theupper bound seek delay prior to playback. Prior to commencement ofplayback the channel conditions may be unknown, and playback commencesat the lowest bitrate (although many embodiments measure the channelrate prior to commencing playback and can utilize bitrate streams havinghigher maximum bitrates). The playback device then measures (104) thechannel data rate and determines (106) whether there has been a changein channel rate. If there has been a change, the playback device chooses(102) the stream that is optimally encoded for the new channel rate. Theprocess repeats until the end of the stream (108) is reached. As canreadily be appreciated, the process illustrated in FIG. 4 is extremelysimple. The playback device can disregard seek delay due to the encodingof the streams ensuring that lower bit rate streams have an upper boundseek delay equal to or less than the upper bound seek delay of thehigher bitrate streams, and due to the playback device buffering aportion of media having a playback duration equal to the upper boundseek delay of the stream. Although a specific process is illustrated inFIG. 4, similar processes appropriate to specific applications can alsobe utilized in accordance with embodiments of the invention including(but not limited to) processes that utilize lower bounds on seek delayto reduce the time taken to commence playback.

Buffering and Playback Using Lower Bound Seek Delay

When video is encoded at a variable bit rate, all pictures do not havethe same size. Depending upon the channel rate, a large frame (e.g. anI-frame) can take more than one frame period to download. Therefore, theclustering of large frames determines the actual minimum seek delay foravoiding stalls in the playback of the video stream at a given channelrate. There is an advantage in knowing the minimum or lower bound seekdelay to reduce the time taken to commence streaming and to make streamswitching decisions when the amount of data in the buffer is less thanthe upper bound seek delay. When the amount of data in the buffer isless than the upper bound seek delay and the need to switch streamsarises, ideally a stream is selected that has a lower bound seek delaythat is smaller than the amount of time it will take to playback thevideo stored within the buffer. Due to the fact that video data istypically requested in chunks, the lower bound seek delay can bedetermined based upon the size of the chunks as follows:

$\begin{matrix}{{Delay} = {\max{\forall{{chunks}\left( {\frac{\sum\limits_{i = 0}^{n}{{size}\mspace{14mu}{chunk}_{i}}}{rate} - {{\left( {n - 1} \right) \cdot {duration}}\mspace{14mu}{chunk}}} \right)}}}} & \lbrack 1\rbrack\end{matrix}$

Effectively, the lower bound seek delay is determined as the maximum ofthe difference for each chunk in the time between the chunk arriving(i.e. the sum of size of all preceding chunks and the current chunk ndivided by the channel rate) and when it should have arrived to preventunderflow (i.e. (n−1)*playback duration of each chunk) The upper boundseek delay is determined based upon the maximum buffer size of stream ofdata as follows:

$\begin{matrix}{{Delay} = \frac{{{level}\mspace{14mu}{buffersize}} + {{level}\mspace{14mu}{{rate} \cdot {duration}}\mspace{14mu}{chunk}}}{rate}} & \lbrack 2\rbrack\end{matrix}$

The upper bound seek delay is determined to be the time it takes todownload the largest possible chunk plus the time taken to playback thechunk. Due to the manner in which variable bit rate video is encoded,the largest possible frame in the stream can theoretically be as largeas the buffer size.

Using either equation, a minimum buffer delay for a stream can bedetermined. While the lower bound seek delay is typically less than theupper bound seek delay, the lower bound seek delay also changesdepending upon the point at which playback commences. The lower boundseek delay for each point in each stream can be calculated in advance bya server and provided to the playback device as required. Alternatively,many H.264 bitstreams carry delay information in the SEI bufferingmessage in the field initial_cpb_removal_delay. While this value is notthe same as performing the above calculation, it has been found to beequivalent and, in several embodiments, the initial_cpb_removal_delayvalue at a particular point in the stream is used as the lower boundseek delay at that point.

Playback can typically commence sooner when a playback device uses thelower bound seek delay to determine the amount of buffered media thatcan prevent underflows. During playback, however, the playback devicemay have less buffered content than when the upper bound seek delay isused. Therefore, when the channel rate decreases, the playback devicehas less time to switch to a lower bitrate stream. In many embodiments,the playback device selects a stream having a maximum bitrate that isless than and closest to the new channel rate, and from which a portionof media having a playback duration equal to the lower bound seek delaycan be downloaded in the time it will take to playback the remainingbuffered media. In many instances, the stream that is optimally encodedfor the channel conditions is not the stream selected, becauseinsufficient media can be downloaded in the time it will take toplayback the buffered content. Instead, a lower bitrate stream may beselected so that the portion of media downloaded during the playback ofthe buffered media has a playback duration equal to or greater than thelower bound seek delay for the lower bitrate stream. When a sufficientportion of media has accumulated in the playback device's buffer, theplayback device can switch to a higher rate bitstream that is optimalfor the channel bitrate.

A playback device utilizing lower bound seek delays when switchingbetween streams in accordance with an embodiment of the invention canobtain the lower bound seek delay for each of the streams from theserver. In several embodiments, the lower bound seek delays are encodedin the stream received from the server. In a number of embodiments, theplayback device separately obtains the lower bound seek delays from theserver. In several embodiments, the playback device utilizes the highestof the lower bound seek delays for each of the streams as the lowerbound seek delay for all of the streams. The highest of the lower boundseek delays can be referred to as the safe seek delay, because it issafe for the playback device to use with respect to all of the streams.Utilizing the safe seek delay instead of the lower bound seek delays foreach stream can be desirable as knowing the safe seek delay of thecurrent stream gives sufficient knowledge over the seek delay of allother alternative streams. In many embodiments, the maximuminitial_cpb_removal_delay value across all corresponding locations inthe streams is considered to be the safe seek delay and is inserted inthe initial_cpb_removal_delay field in the SEI buffering message of eachstream. In other embodiments, any of a variety of techniques is utilizedto provide the playback device with the delay associated with each ofthe streams in advance of the playback device selecting a new stream inresponse to a change in channel rate.

Buffering and Playback Using a Combination of Lower and Upper Bound SeekDelay

Utilizing lower bound seek times enables playback to commence morerapidly. Using upper bound seek times in combination with appropriatelyencoded content simplifies switching between streams. In a number ofembodiments, playback devices utilize processes that harness thebenefits of both approaches.

A process for performing adaptive bitrate streaming in accordance withan embodiment of the invention is illustrated in FIG. 5. The process 200assumes that the streams are encoded in a similar manner to the streamsdescribed above with respect to the process shown in FIG. 4 and thatlower bound seek delay or safe seek delay information is available tothe playback device (e.g. within each stream). The process 200 commencesin a similar manner to the adaptive bitrate streaming process describedabove that utilizes a lower bound seek delay or a safe seek delay todetermine when to commence playback. The adaptive bitrate processdiffers with respect to the process described above, however, in thatthe rates of the streams selected during the initial adaptive bitrateprocess (202) are a predetermined amount less than the channel rate.Lower bitrate streams are selected so that media accumulates in theplayback device's buffer. In a number of embodiments, a stream isselected having a bitrate that is 33% less than the channel rate.Although in other embodiments the predetermined amount can be greater,less, and/or determined in other ways that result in the accumulation ofmedia in the buffer of the playback device during media playback. Whenthe amount of media accumulated in the buffer is determined (204) tohave a playback duration greater than or equal to the upper bound seekdelay of the stream and/or a size greater than or equal to the maximumbuffer size for the encoded stream, the adaptive bitrate streamingprocess transitions (206) to the adaptive bitrate streaming processillustrated in FIG. 4 in which the combination of the encoding of thestreams and the amount of media stored in the playback device's bufferenables the playback device to select the optimally encoded stream for agiven channel rate without regard to seek delay.

Although a specific process is illustrated in FIG. 5 and specific streamswitching rules are discussed above combining the benefits of lowerbound seek times to achieve rapid playback and the simplicity of upperbound seek times when switching streams, a number of processes canutilize varying buffering requirements during adaptive bitrate streamingto achieve different performance objectives in accordance withembodiments of the invention.

Stream Switching when Using a Combination of Lower and Upper Bound SeekDelay

When a playback device can utilize both lower and upper bound seekdelays, sets of stream switching rules are utilized that are dependentupon the amount of buffered video. As discussed above with respect toFIG. 5, a playback device can utilize lower bound seek delays to reducethe time spent buffering prior to commencing playback of video. Duringthis initial start up period, a stream switching rule can be appliedthat involves selecting a stream where the lower bound seek delay of thestream is less than the amount of time it will take to playback thevideo buffered at the time of the stream switch. Such a rule can beexpressed as follows:

-   -   1. If (buffer level<upper bound seek delay) then select a stream        having a lower bound seek delay that is less than the playback        duration of the buffered video.

In the event that a network bandwidth measurement cannot be reliablymade, a playback device can utilize an additional streaming rule thatforces the selection of the lowest bitrate stream. This stream switchingrule can be expressed as follows:

-   -   2. If (buffer level<upper bound seek delay and network data rate        is unreliable or inaccurate) then select the lowest bitrate        stream.

When a steady network data rate is experienced for a long enough periodof time or a lower bitrate stream is selected to grow the buffer in themanner described above with respect to FIG. 5, then the amount of videobuffered by the playback device will climb to the point where it exceedsthe upper bound seek delay of the stream. At which point, streamswitching can be performed based upon the measured network bandwidth andthe maximum bitrate of the streams. Where the upper bound seek delay ofstreams encoded at a lower maximum bitrate are known, a similar streamswitching decision could be made based upon the upper bound seek delayof the stream to which the switch is being made. The correspondingstream switching rule can be expressed as follows:

-   -   3. If (network data rate is reliable and upper bound seek        delay<=buffered video), then select the stream having a maximum        bitrate that is the highest of the alternative streams that have        a maximum bitrate less than the network data rate.

When the playback device is downloading chunks of a stream much fasterthan expected according to the level rate, then the channel data rate ismuch higher than the maximum bitrate of the stream and it is difficultto reliably measure the network data rate so a higher bitrate stream canbe selected once the amount of video buffered exceeds the upper boundseek delay. This stream switching rule can be expressed as follows:

-   -   4. If (expected time to download chunk>>actual time taken to        download chunk and upper bound seek delay<=buffered video), then        select the stream having the next highest maximum bitrate.

In addition to the above rules, playback devices in accordance withembodiments of the invention can incorporate hysteresis in the streamswitching rules to prevent excessive switching of levels when themeasured network data rate oscillates around a decision threshold.Furthermore, a playback device may include safety margins or headroombetween the measured network data rate and the maximum bitrate of aselected stream when selecting streams. In many embodiments, theheadroom can be temporarily increased after a significant bandwidth drop(e.g. a predetermined amount or percentage) to reduce the likelihood ofbuffer underflow in the event of an additional drop in network datarate. Also, the playback device can throttle the downloading of othermedia (particularly at start up) to reduce the time taken for theplayback duration of the buffered video data for a particular stream tomatch or exceed the upper bound seek delay.

Although specific stream switching rules are discussed above combiningthe benefits of lower bound seek times to achieve rapid playback and thesimplicity of upper bound seek times when switching streams, any of avariety and combination of stream switching rules can be utilized duringadaptive bitrate streaming to achieve different performance objectivesin accordance with embodiments of the invention.

Although the present invention has been described in certain specificaspects, many additional modifications and variations would be apparentto those skilled in the art. It is therefore to be understood that thepresent invention may be practiced otherwise than specificallydescribed, including various changes in the implementation such asutilizing encoders and decoders that support features beyond thosespecified within a particular standard with which they comply, withoutdeparting from the scope and spirit of the present invention. Thus,embodiments of the present invention should be considered in allrespects as illustrative and not restrictive.

What is claimed is:
 1. A source encoder configured to encode sourcevideo as a number of alternative video streams, the source encodercomprising: memory; and a processor configured by a source encodingapplication to: receive multimedia content, where the multimedia contentcomprises source video data having a primary resolution and a primarysample aspect ratio; and encode the source video data as a set ofalternative video streams, where: the alternative video streams havedifferent maximum bitrates; and the alternative video streams areencoded to have an upper bound seek delay that is equal to or less thanthe upper bound seek delay of streams in the set of alternative videostreams that are encoded at a higher maximum bitrate where the upperseek bound delay of a particular one of the alternative video streams isdetermined as follows:${Delay} = \frac{{{level}\mspace{14mu}{buffersize}} + {{level}\mspace{14mu}{{rate} \cdot {duration}}\mspace{14mu}{chunk}}}{rate}$where level buffersize is the maximum buffer size constraint used inencoding the video stream; duration chunk is the time taken to playbacka single chunk of video of the particular one of the alternative videostreams; level rate is the maximum bitrate of the particular one of thealternative video streams; and rate is a network data rate.
 2. Thesource encoder of claim 1, wherein: the source encoding applicationfurther configures the processor to encode the alternative video streamsat predetermined maximum bitrates and maximum buffer sizes; and theupper bound seek delay of a stream is determined by dividing thepredetermined maximum buffer size of the stream by the predeterminedmaximum bitrate of the stream.
 3. The source encoder of claim 1, whereinthe source encoding application further configures the processor toencode the alternative video streams so that each stream in the set ofalternative streams has the same aspect ratio.
 4. The source encoder ofclaim 1, wherein the source encoding application further configures theprocessor to determine a lower bound seek delay for each stream in theset of alternative video streams.
 5. The source encoder of claim 4,wherein the source encoding application further configures the processorto include within each stream its lower bound seek delay.
 6. The sourceencoder of claim 4, wherein the source encoding application furtherconfigures the processor to: determine the alternative stream that hasthe highest lower bound seek delay; and include the highest lower boundseek delay within each stream.
 7. The source encoder of claim 6, whereinthe alternative video streams are encoded in accordance with the H.264standard.
 8. The source encoder of claim 7, wherein the source encodingapplication further configures the processor to include the highestlower bound seek delay in a initial_cpb_removal_delay field in an SEIbuffering message within each stream.
 9. The source encoder of claim 4,wherein the source encoding application configures the processor todetermine the lower bound seek delay of a particular one of thealternative streams as follows:${Delay} = {\max{\forall{{chunks}\left( {\frac{\sum\limits_{i = 0}^{n}{{size}\mspace{14mu}{chunk}_{i}}}{rate} - {{\left( {n - 1} \right) \cdot {duration}}\mspace{14mu}{chunk}}} \right)}}}$where: size chunk_(i) is a size of a particular chunk, i, in theparticular one of the alternative streams, where rate is the networkdata rate, and duration chunk is the time taken to playback a singlechunk of video of the particular one of the alternative video streams.10. A method of encoding source video as a number of alternative videostreams, comprising: receiving multimedia content, where the multimediacontent comprises source video data having a primary resolution and aprimary sample aspect ratio using a source encoder; encoding the sourcevideo data as a set of alternative video streams using the sourceencoder, where: the alternative video streams have different maximumbitrates; and the alternative video streams are encoded to have an upperbound seek delay that is equal to or less than the upper bound seekdelay of streams in the set of alternative video streams that areencoded at a higher maximum bitrate where the upper seek bound delay ofa particular one of the alternative video streams is determined asfollows:${Delay} = \frac{{{level}\mspace{14mu}{buffersize}} + {{level}\mspace{14mu}{{rate} \cdot {duration}}\mspace{14mu}{chunk}}}{rate}$where level buffersize is the maximum buffer size constraint used inencoding the video stream; duration chunk is the time taken to playbacka single chunk of video of the particular one of the alternative videostreams; level rate is the maximum bitrate of the particular one of thealternative video streams; and rate is a network data rate.
 11. Themethod of claim 10, further comprising: encoding the alternative videostreams at predetermined maximum bitrates and maximum buffer sizes usingthe source encoder; wherein the upper bound seek delay of a stream isdetermined by dividing the predetermined maximum buffer size of thestream by the predetermined maximum bitrate of the stream.
 12. Themethod of claim 10, further comprising encoding the alternative videostreams so that each stream in the set of alternative streams has thesame aspect ratio.
 13. The method of claim 10, further comprisingdetermining a lower bound seek delay for each stream in the set ofalternative video streams.
 14. The method of claim 13, furthercomprising including within each stream its lower bound seek delay. 15.The method of claim 13, further comprising: determining the alternativestream that has the highest lower bound seek delay; and including thehighest lower bound seek delay within each stream.
 16. The method ofclaim 15, wherein the alternative video streams are encoded inaccordance with the H.264 standard.
 17. The method of claim 16, furthercomprising including the highest lower bound seek delay in ainitial_cpb_removal_delay field in an SEI buffering message within eachstream.
 18. The method of claim 14, further comprising determining thelower bound seek delay as follows:${Delay} = {\max\;{\forall{{chunks}\left( {\frac{\sum\limits_{i = 0}^{n}{{size}\mspace{14mu}{chunk}_{i}}}{rate} - {{\left( {n - 1} \right) \cdot {duration}}\mspace{14mu}{chunk}}} \right)}}}$where: size chunk_(i) is a size of a particular chunk, i, in theparticular one of the alternative streams, where rate is the networkdata rate, and duration chunk is the time taken to playback a singlechunk of video of the particular one of the alternative video streams.19. A non-transitory machine readable medium containing processorinstructions, where execution of the instructions by the processorcauses the processor to perform a process comprising: receivingmultimedia content, where the multimedia content comprises source videodata having a primary resolution and a primary sample aspect ratio; andencoding the source video data as a set of alternative video streams,where: the alternative video streams have different maximum bitrates;and the alternative video streams are encoded to have an upper boundseek delay that is equal to or less than the upper bound seek delay ofstreams in the set of alternative video streams that are encoded at ahigher maximum bitrate where the upper seek bound of a particular one ofthe alternative video streams is determined as follows:${Delay} = \frac{{{level}\mspace{14mu}{buffersize}} + {{level}\mspace{14mu}{{rate} \cdot {duration}}\mspace{14mu}{chunk}}}{rate}$where level buffersize is the maximum buffer size constraint used inencoding the video stream; duration chunk is the time taken to playbacka single chunk of video of the particular one of the alternative videostreams; level rate is the maximum bitrate of the particular one of thealternative video streams; and rate is a network data rate.